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Introduction 


The back-propagation algorithm for training neural networks is used in a variety of 
applications due to its simplicity and ease of implementation. This paper assumes the 
reader is somewhat familar with neural networks in general and the back-propagation 
algorithm in particular. An introduction to these topics can be found in [Rume 86]. The 
simplicity and widespread use of the back-propagation algorithm make it a logical starting 
point for developing a parallel algorithm for training neural networks. The work presented 
in this paper focuses on two back-propagation based approaches to training a neural 
network in parallel. 

The first approach is a pattern partitioning approach. In this approach, the entire 
network is represented on each processor and the set of input patterns is divided among the 
available processors. Each processor independently computes the weight changes 
necessary to minimize the error for its subset of the input patterns. After each processor 
has computed these weight changes, they are combined and broadcast to all processors. 

The second approach involves partitioning the neurons of the neural network among the 
available processors and having each processor propagate the entire set of input patterns. 
The advantage of this approach is that changes do not need to be combined among the 
processors. However, communication is still needed during the propagation phase since 
each processor now represents only a portion of the entire network. 

The pattern partitioning approach was implemented first due to its simplicity. Given n 
input patterns and p processors, each processor was given n/p input patterns (with 
appropriate rounding for non-integer values, of course) and the entire set of weights for the 
network. During each epoch, each processor propagates its input patterns through the 
network, and then back-propagates the error values. At some point, the processors need to 
combine the delta values resulting from the back-propagation phase. 

Two different approaches to this combination were attempted. The first involved 
saving the results of the pattern propagation and the delta values from the back-propagation 
in matrices and then combining these matrices. This turned out to be a naive approach 
since the amount of communication that needed to be done increased linearly with the 
number of input patterns. A better approach was to collect the weight delta values for all of 
a processor's nip patterns and then to combine these weight delta matrices by summing 
them. This approach provided much better scalability. 

The network-partitioned approach that was implemented involves dividing the e 
network elements among the p processors. The model of a software pipeline is used, with 
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the processors communicating in a ring fashion. Patterns are placed in one end of the 
pipeline, and as they flow through the pipeline are processed by each processor's neurons. 
One processor does no actual processing of neurons but acts as a monitoring processor, 
placing messages into the pipeline, retrieving them from the end of the pipeline and keeping 
track of bookeeping tasks. The network elements are divided among the other p-l 
processors. 

Measures of Efficiency 

The performance of a parallel algorithm can be split into two components: the time 
spent doing useful work, and the time spent idling or in communications. Call these two 
components r c ^, the time spent doing useful calculations, and tJ omm , the time spent in 
communcations and not performing useful work (the superscript i denotes the processor 
number). The sum of for all processors is defined as T ca i c , and the sum of tJ omm for 
all processors isT comm . The total time taken by an algorithm on p processors is: 

P 

T — ^ (t ca j c + fcomm) = T ca lc + T comm ~ pTp 
i =1 

where Tp is the execution time on p processors. 

Problem Size W: The problem size is defined as the total amount of computation done 
by the best known sequential algorithm. 

Speedup S : The speedup on p processors is defined as ratio of where T l is the time 

Tp 

of the best sequential algorithm. 

Efficiency E: Efficiency measures the effective utilization of p processors and is defined 
S p 

as —. Essentially measures what percentage of a linear speedup was obtained on p 
processors. 

Iso-efficiency - The iso-efficiency function describes the relationship between W and p 
that must be maintained in order to keep a constant efficiency. See [Gupt 1990] for a 
complete discussion. 

Definitions 

In order to analyze the efficiency of these algorithms, we need some common definitions. 
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The inputs to a neural network consist of: 


n input patterns, and 

m neuron layers, each of size k, along with (m - 1) weight matrices of size kxk 
connecting the layers. Note that we are assuming that each layer has the same 
number of neurons for the purposes of our analysis. The programs that were 
devloped allow differing number of neurons for each layer. This is generally the 
case in actual neural network applications. 

We are given p processors. 

From these values, we can derive some useful expressions: 

M, the sizes of all weight matrices, is (m - 1 )k 2 . 

C , the number of hidden and ouput layer neurons, is (m - 1 )k. 

Pattern-Partitioned Approach 

The pattern partitioned approach on a hypercube is fairly simple to implement. The 
host processor send the weight matrices for the entire network to each processor, along 
with the subset of n/p input patterns that each processor will be handling. During an 
epoch, each processor calculates the weight delta values for its pattern set by propagating 
its inputs through the network. These weight delta matrices are then sent up a tree (mapped 
onto a hypercube in the usual manner). At each non-leaf node, the two sets of matrices are 
combined using matrix addition. The node at the root of the tree then has the weight delta 
matrices for the entire input pattern set, and broadcasts these to all the other processors (in 
log time, using the typical hypercube broadcasting method). Each processor then updates 
its weights and begins processing its input patterns for the next epoch. 


The sequential algorithm takes time proportional to M for each input pattern. Thus, the 
sequential algorithm's time per epoch is O (nM). 
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Analysis 


The computation time per processor is: 

t calc 

The communication time per processor is: 
tcomm-M log p 

Thus, the speedup of the pattern-partitioned approach is: 

S =_ nM. _ 

n y- + M log p 

The efficiency is: 

E _ _- 1 

nM + M p log p ^ + p log p 

n 

In order to maintain constant efficiency, P must remain constant. Thus, the iso¬ 
efficiency function is: 
n oc p log p 
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Experimental Results 


Several different sets of input data were generated in order to evaluate the performance 
of the pattern-partitioned approach. The intuitive result that better speedups would be 
obtained with larger numbers of input patterns held. Speedup curves for three of these 
input sets are depicted below: 


Speedup Curves for Pattern-Partitioned Approach 



Fig. 1. Speedup curves for pattern-partitioned approach. The small network contained 3 
layers with 9, 4 and 2 neurons. The large network contained 3 layers with 10,21 and 10 
neurons. 
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Pipelined Approach 


The second method that was implemented involved a network-partitioning, rather than a 
pattern-partitioning approach approach. During an epoch, a given pattern will be handled 
by every processor in the system, but that processor will only do a portion of the 
processing for that pattern. Only hidden and output layer nodes are represented on 
processors; since the input layer has no weights incident on it, any processing of input 
layer nodes is assumed to be handled by the first hidden layer. One processor is assigned 
the role of manager and is responsible for filling the pipeline with messages, collecting the 
output at the other end of the pipeline, and various bookeeping tasks. The C hidden and 
output layer neurons are evenly divided among the remaining p -1 processors 1 . 

Initially, the managing processor generates messages to completely fill the pipeline; 
thereafter, it receives output from the end of the pipeline and continues to ensure that the 
pipeline is full. For a complete discussion of the algorithm, see [Alle 89]. 

Each message in this approach contains the input pattern, target output pattern, a 
placeholder for the input to and output from the current layer, and a set of m vectors to 
hold the error information for each layer. The size of each message is O (mk). 

Analysis 

The computation time per processor per epoch is: 

tcaic=%M 

The communication time per processor epoch is: 

^comm ~ HTTlk 

Thus, the speedup of the pipelined network-partitioned approach is: 

S - nmk 2 

nmkl+nmk 

P 


1 Allen and Saha [ALLE 89] conducted experiments which showed that output layer neurons take 3 times 
as long to process as hidden layer neurons, so some load balancing may be necessary to offset this 
difference. However, this was not taken into account in this implementation. 
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The efficiency is: 

E - nmk 2 

nmk 2 + pnmk 


1 

1 + 2 - 

k 


In order to maintain constant efficiency, 2. must remain constant. 

k 

function is: 


Thus, the iso-efficiency 


k oc p 


Experimental Results 

Several neural networks of various sizes were trained using this pipelined approach. 
Due to a limitation on message sizes between processors on the NCUBE multicomputer, 
results were only analyzed up to 32 processors. The calculations do not include the 
managing processor, since its functions could also be handled by the host processor. 
Thus, on a 3-dimensional hypercube, 7 processors were performing training, so the 
speedup is presented for 7 processors, not 8 processors. On the graph below, there are 
two curves: one is for the pipelined approach of a moderate sized network and the other is 
for the same inputs under the pattern-partitioned approach described above. Speedup 
curves for both methods are shown. 

Speedup Data for Pipelined Approach 



Fig 2. Data for large network dataset (3 layers, 10,21 and 10 neurons per layer) for 
pipelined and pattern-partitioned approaches. 
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Discussion 


Both approaches seemed to perform fairly well. The pattern-partitioned approach is 
less scalable than the pipelined approach, but performs better with a smaller number of 
processors. The pipelined approach could be improvemed in several ways. One way 
would be to balance the processing times taken for each processor in the pipeline by 
performing some analysis on the time taken for hidden and output layer neurons rather than 
simply dividing the available neurons evenly among the processors. This would eliminate 
any botdenecks caused by differences in processor speed, and would increase concurrency. 
Another possibility is to have a two-way pipeline, in which messages flow to the end of the 
network, and then flow backwards to implement the back-propagation phase rather than 
saving the results of the previous epoch to be cycled through in the next epoch. This 
would reduce the message size by a factor of m and would undoubtedly increase the 
efficiency of the algorithm. 
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end of iteration loop 
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VectorAllocate (iv, vsize); 

step “ (hi - lo) / vsize; void LandN (ISize, num, 

for (1 - 0, y ■= lo; i < vsize; i++, y+=step) int ISize[]; 

v(l] - 1.0 / (1.0 + exp (-y)); int num; 

doneinit = TRUE; int *L, *N; 
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mtype = M_WUPDATE; 

nread ({char »)mbuf, sizeof (UPDATEMSG), sleft, Smtype, sflag); 






